There has been an explosion of employing visual assist technologies in automotive application in recent years. Cameras are now common features in vehicles to assist drivers in performing their tasks. Furthermore, an integrated visual system would be essential in advancements of autonomous vehicles. Accordingly, there is an increasing need to improve vehicle visual systems for seamless display of the surroundings in real-time; to optimize image analytics and decision making processes; and to minimize time and efforts required for maintenance and calibration of the visual systems.
Exploiting Ultra Wide-Angle (UWA) lenses, having Field of Views (FOV) of at least 180°, may be part of a solution for these challenges when they are complimented with image and video processing capabilities. A powerful image and video processor could provide fast and flexible processing of the acquired images. Further, it could eliminate installment of expensive and bulky optics in the vehicle. Yet further, adaptive transformations may enable the driver or other passengers (collectively referred to as the users) to select areas of interest to view and receive a distortion corrected image in real-time. The co-pending patent application PCT/US2012/027189 teaches an innovative method for unfolding images acquired by Wide-Angle (WA) and Ultra Wide-Angle lenses. The disclosed techniques are based on specialized transformations, including a Zero-Content-Loss (ZCL) transformation that may be applied to the visually distorted images captured through a UWA lens. Accordingly, a perspective corrected image may be obtained with a high degree of accuracy for areas of interest, yet all the information contained in the original captured scene is maintained. A localized and content adaptive correction may further be applied to selected regions of interest.
The present invention discloses novel vehicle vision system architectures and methods of implementation based on proprietary image processing means. A full rear and peripheral view input may be provided with minimum number of cameras, where ZCL imagery guarantees no blind spots for the driver. Optionally and preferably, adding a front camera may provide a 360° image acquisition capability. Such a multi-camera (also referred to as MC from here on) systems may realize various innovations in the automotive industry. For example, the rear-view and/or side-view mirrors and the task of adjusting them, either manually or electrically may be replaced by a minimum number of smart cameras. Alternatively, one could envision operating a mirror-less vehicle where all mirrors are replaced by displays. The latter would be essential for autonomous driving where decision makings would be done by the machine instead of the driver. Further, such visual systems may be integrated with alarm systems, tracking systems, communication systems, etc. to facilitate more features such as recording and communicating the events on the road.
In a vision system, equally as important as the capture devices, are the display devices. That may be one or more LCD displays, one or more projectors, one or more Head-Up-Displays (HUDs), or some combination of them. An issue with LCD displays may be their physical positioning and plane of focus that are necessarily away from the driver's front field of view and road focused focal point. This means looking away and changing focus in order to see the LCD display, which can potentially be very dangerous, even if the distraction lasts only a few seconds. For these reasons, we are seeing rapid growth of HUD displays for vehicles, which present information in the form of a virtual image displayed a few meters in front of the driver, avoiding having to look away or lose focus. The image is also appropriately positioned not to obstruct the normal view. Because HUD displays are based on projection optics, and the windshield functions in itself as an optical element, this leads to various geometric distortions. Just as a processor corrects for distortion of an UWA lens, the same type of processor can correct for distortions on the display side for a HUD. These corrections may be combined with any user specific corrections for customization of their view. Furthermore, analogous to multiple cameras, multiple displays (HUDs and LCD displays) may be used to present information in novel ways, that are more natural to the driver experience. Multiple HUD displays may be blended and stitched together to provide large view displays that can be very useful for applications such as augmented reality for safety, highlighting objects, displaying map information or even detailed graphics/video content when the vehicle is in park, etc. A 360° view discussed above may be presented on a large view HUD made from two or more smaller HUDs in sufficient detail and correct perspective. Combining multiple HUD displays into single novel views also requires geometric correction, which can be facilitated by the same processor used for correcting single HUD distortions. The present invention as part of its vision system includes multiple display systems, functioning independently or jointly, to present novel sophisticated views to the driver. All prior art vision based systems have solely focused on the capture process, with no consideration of the display side.
This invention benefits teachings of the co-pending application PCT/US2012/027189; the content of which are incorporated by reference in their entirety. All methods and transformations may be implemented by software or in real-time using proprietary hardware implementation of the transformations, for example as described in U.S. Pat. Nos. 7,324,706 and 8,055,070. The system architectures are embodied in a vehicle environment but may be applicable to other environments with similar video capture and display settings.